Monitoring distributed web application transactions

ABSTRACT

The present technology monitors a web application provided by one or more services. A service may be provided by applications executing on a virtual machine. The monitoring system provides end-to-end business transaction visibility, identifies performance issues quickly and has dynamical scaling capability across monitored systems including cloud systems, virtual systems and physical infrastructures. Agents may be installed on one or more application servers at an application level, virtual machine level, or other level. An agent may monitor the virtual machine, application and application communications, may communicate with a controller, and may provide monitoring data to the controller. The controller may evaluate the performance of the application and/or virtual machine, model service and/or application flow, and determine information regarding distributed service and/or application performance. The monitoring technology determines how each service and/or application portion is operating, establishes a baseband for operation, and determines the architecture of the distributed system.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the priority benefit of U.S. ProvisionalApplication Ser. No. 61/241,256, titled “Automated Monitoring ofBusiness Transactions,” filed Sep. 10, 2009, the disclosure of which isincorporated herein by reference.

BACKGROUND OF THE INVENTION

The World Wide Web has expanded to provide web services faster toconsumers. Web services may be provided by a web application which usesone or more services to handle a transaction. The applications may bedistributed over several machines, making the topology of the machinesthat provides the service more difficult to track and monitor.

Monitoring a web application helps to provide insight regarding bottlenecks in communication, communication failures and other informationregarding performance of the services the provide the web application.When a web application is distributed over several machines, trackingthe performance of the web service can become impractical with largeamounts of data collected from each machine.

There is a need in the art for web service monitoring which mayaccurately and efficiently monitor the performance of distributedapplications which provide a web service.

SUMMARY OF THE CLAIMED INVENTION

The present technology monitors a network or web application provided byone or more distributed network services. The monitoring system maymonitor distributed web applications across a variety ofinfrastructures. The system is easy to deploy and provides end-to-endbusiness transaction visibility. The monitoring system may identifyperformance issues quickly and has a dynamical scaling capability acrossa monitored system. The present monitoring technology has a lowfootprint and may be used with cloud systems, virtual systems andphysical infrastructures.

Agents may be installed on one or more servers at an application level,virtual machine level, or other level. An agent may monitor acorresponding application and application communications. The webapplication may consist of one or more services implemented by a virtualmachine, or an application within a virtual machine, on an applicationserver. Each agent may communicate with a controller and providemonitoring data to the controller. The controller may process the datato evaluate the performance of the application, model the flow of theweb application, and determine information regarding distributed webapplication performance. The monitoring technology determines how eachdistributed application portion is operating, establishes a baseband foroperation, and determines the architecture of the distributed system.

The present technology may monitor a distributed application thatperforms one or more business transactions. Agents may communicate withcode within an application that monitors calls and requests received andsent by an application. By monitoring incoming and outgoing calls andrequests, and by monitoring the performance of services (virtualmachine) that process the incoming and outgoing request, the presenttechnology may determine the performance and structure of complicatedand distributed business transactions.

Monitoring a business transaction may include associating a requestreceived by an application with a thread of an application. A call maybe modified with monitoring parameters by the application, wherein thecall may be determined to be associated with the thread. Runtime datathat includes the monitoring parameters may and is associated with thecall may be reported to a controller.

A controller may receive runtime data from a plurality of servers. Amapping of the plurality of servers may be constructed based on theruntime data. Performance data may be determined for each of theplurality of servers based on the runtime data.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an exemplary system for monitoring businesstransactions.

FIG. 2 is a block diagram of an exemplary application server.

FIG. 3 is a flowchart of an exemplary method for monitoring businesstransactions.

FIG. 4 is a flowchart of an exemplary method for associating a requestwith a thread.

FIG. 5 is a flowchart of an exemplary method for processing a receivedrequest.

FIG. 6 is a flowchart of an exemplary method for generating a call.

FIG. 7A is a flowchart of an exemplary method for responding to areceived request.

FIG. 7B is a flowchart of an exemplary method for reporting runtime datato a controller.

FIG. 8 is a flowchart of an exemplary method for controlling businesstransaction monitoring.

FIG. 9 is an exemplary interface for reporting monitoring data forbusiness transactions.

FIG. 10 is an exemplary interface for viewing monitoring data forbusiness transactions.

FIG. 11 is a block diagram of an exemplary computing device.

DETAILED DESCRIPTION

The present technology monitors a network or web application provided byone or more distributed applications. The web application may beprovided by one or more web services each implemented as a virtualmachine or one or more applications implemented on a virtual machine.Agents may be installed on one or more servers at an application level,virtual machine level, or other level. An agent may monitor acorresponding application (or virtual machine) and applicationcommunications. Each agent may communicate with a controller and providemonitoring data to the controller. The controller may process the datato evaluate the performance of the application or virtual machine, modelthe flow of the application, and determine information regarding thedistributed web application performance. The monitoring technologydetermines how each distributed web application portion is operating,establishes a baseband for operation, and determines the architecture ofthe distributed system.

The monitoring system may monitor distributed web applications across avariety of infrastructures. The system is easy to deploy and providesend-to-end business transaction visibility. The monitoring system mayidentify performance issues quickly and has a dynamical scalingcapability across a monitored system. The present monitoring technologyhas a low footprint and may be used with cloud systems, virtual systemsand physical infrastructures.

The present technology may monitor a distributed web application thatperforms one or more business transactions. A business transaction maybe a set of tasks performed by one or more distributed web applicationsin the course of a service provide over a network. In an e-commerceservice, a business transaction may be “add to cart” or “check-out”transactions performed by the distributed application.

Agents may communicate with code within virtual machine or anapplication. The code may detect when an application entry point iscalled and when an application exit point is called. An applicationentry point may include a call received by the application. Anapplication exit point may include a call made by the application toanother application, virtual machine, server, or some other entity. Thecode within the application may insert information into an outgoing callor request (exit point) and detect information contained in a receivedcall or request (entry point). By monitoring incoming and outgoing callsand requests, and by monitoring the performance of a local applicationthat processes the incoming and outgoing request, the present technologymay determine the performance and structure of complicated anddistributed business transactions.

FIG. 1 is a block diagram of an exemplary system 100 for monitoringbusiness transactions. System 100 of FIG. 1 includes client device 105,mobile device 115, network 120, network server 125, application servers130, 140, 150 and 160, asynchronous network machine 170, data stores 180and 185, and controller 190.

Client device 105 may include network browser 110 and be implemented asa computing device, such as for example a laptop, desktop, workstation,or some other computing device. Network browser 110 may be a clientapplication for viewing content provided by an application server, suchas application server 130 via network server 125 over network 120.Mobile device 115 is connected to network 120 and may be implemented asa portable device suitable for receiving content over a network, such asfor example a mobile phone, smart phone, or other portable device. Bothclient device 105 and mobile device 115 may include hardware and/orsoftware configured to access a web service provided by network server125.

Network 120 may facilitate communication of data between differentservers, devices and machines. The network may be implemented as aprivate network, public network, intranet, the Internet, or acombination of these networks.

Network server 125 is connected to network 120 and may receive andprocess requests received over network 120. Network server 125 may beimplemented as one or more servers implementing a network service. Whennetwork 120 is the Internet, network server 125 maybe implemented as aweb server.

Application server 130 communicates with network server 125, applicationservers 140 and 150, controller 190. Application server 130 may alsocommunicate with other machines and devices (not illustrated in FIG. 1).Application server 130 may host an application or portions of adistributed application and include a virtual machine 132, agent 134,and other software modules. Application server 130 may be implemented asone server or multiple servers as illustrated in FIG. 1.

Virtual machine 132 may be implemented by code running on one or moreapplication servers. The code may implement computer programs, modulesand data structures to implement a virtual machine mode for executingprograms and applications. In some embodiments, more than one virtualmachine 132 may execute on an application server 130. A virtual machinemay be implemented as a Java Virtual Machine (JVM). Virtual machine 132may perform all or a portion of a business transaction performed byapplication servers comprising system 100. A virtual machine may beconsidered one of several services that implement a web service.

Virtual machine 132 may be instrumented using byte code insertion, orbyte code instrumentation, to modify the object code of the virtualmachine. The instrumented object code may include code used to detectcalls received by virtual machine 132, calls sent by virtual machine132, and communicate with agent 134 during execution of an applicationon virtual machine 132. Alternatively, other code may be byte codeinstrumented, such as code comprising an application which executeswithin virtual machine 132 or an application which may be executed onapplication server 130 and outside virtual machine 132.

Agent 134 on application server 130 may be installed on applicationserver 130 by instrumentation of object code, downloading theapplication to the server, or in some other manner. Agent 134 may beexecuted to monitor application server 130, monitor virtual machine 132,and communicate with byte instrumented code on application server 130,virtual machine 132 or another application on application server 130.Agent 134 may detect operations such as receiving calls and sendingrequests by application server 130 and virtual machine 132. Agent 134may receive data from instrumented code of the virtual machine 132,process the data and transmit the data to controller 190. Agent 134 mayperform other operations related to monitoring virtual machine 132 andapplication server 130 as discussed herein. For example, agent 134 mayidentify other applications, share business transaction data, aggregatedetected runtime data, and other operations.

Each of application servers 140, 150 and 160 may include an applicationand an agent. Each application may run on the corresponding applicationserver or a virtual machine. Each of virtual machines 142, 152 and 162on application servers 140-160 may operate similarly to virtual machine132 and host one or more applications which perform at lease a portionof a distributed business transaction. Agents 144, 154 and 164 maymonitor the virtual machines 142-162, collect and process data atruntime of the virtual machines, and communicate with controller 190.The virtual machines 132, 142, 152 and 162 may communicate with eachother as part of performing a distributed transaction. In particulareach virtual machine may call any application or method of anothervirtual machine.

Controller 190 may control and manage monitoring of businesstransactions distributed over application servers 130-160. Controller190 may receive runtime data from each of agents 134-164, associateportions of business transaction data, communicate with agents toconfigure collection of runtime data, and provide performance data andreporting through an interface. The interface may be viewed as aweb-based interface viewable by mobile device 115, client device 105, orsome other device. In some embodiments, a client device 192 may directlycommunicate with controller 190 to view an interface for monitoringdata.

Asynchronous network machine 170 may engage in asynchronouscommunications with one or more application servers, such as applicationserver 150 and 160. For example, application server 150 may transmitseveral calls or messages to an asynchronous network machine. Ratherthan communicate back to application server 150, the asynchronousnetwork machine may process the messages and eventually provide aresponse, such as a processed message, to application server 160.Because there is no return message from the asynchronous network machineto application server 150, the communications between them areasynchronous.

Data stores 180 and 185 may each be accessed by application servers suchas application server 150. Data store 185 may also be accessed byapplication server 150. Each of data stores 180 and 185 may store data,process data, and return queries received from an application server.Each of data stores 180 and 185 may or may not include an agent.

FIG. 2 is a block diagram of an exemplary application server 200. Theapplication server in FIG. 2 provides more information for eachapplication server of system 100 in FIG. 1. Application server 200 ofFIG. 2 includes a virtual machine 210, application 220 executing on thevirtual machine, and agent 230. Virtual machine 210 may be implementedby programs and/or hardware. For example, virtual machine 134 may beimplemented as a JAVA virtual machine. Application 220 may execute onvirtual machine 210 and may implement at least a portion of adistributed application performed by application servers 130-160.Application server 200, virtual machine 210 and agent 230 may be used toimplement any application server, virtual machine and agent of a systemsuch as that illustrated in FIG. 1.

Application server 200 and application 220 can be instrumented via bytecode instrumentation at exit and entry points. An entry point may be amethod or module that accepts a call to application 220, virtual machine210, or application server 200. An exit point is a module or programthat makes a call to another application or application server. Asillustrated in FIG. 2, an application server 200 can have byte codeinstrumented entry points 240 and byte code instrumented exit points260. Similarly, an application 220 can have byte code instrumentationentry points 250 and byte code instrumentation exit points 270. Forexample, the exit points may include calls to JDBC, JMS, HTTP, SOAP, andRMI. Instrumented entry points may receive calls associated with theseprotocols as well.

Agent 230 may be one or more programs that receive information from anentry point or exit point. Agent 230 may process the receivedinformation, may retrieve, modify and remove information associated witha thread, may access, retrieve and modify information for a sent orreceived call, and may communicate with a controller 190. Agent 230 maybe implemented outside virtual machine 210, within virtual machine 210,and within application 220, or a combination of these.

FIG. 3 is a flowchart of an exemplary method for monitoring businesstransactions. In some embodiments, the method of FIG. 3 can be performedat any of application servers 130, 140, 150 and 160. Operation ofcontroller 190 is discussed in more detail with respect to FIG. 8.

Entry points and exit points are instrumented using byte codeinstrumentation at step 310. The entry and exit points may beinstrumented in an application residing on an application sever. Theentry and exit points may also be instrumented in a virtual machineresiding on an application sever. Instrumented exit points may includecode that implements a call or request by an application or applicationserver, such as to JDBC, JMS, HTTP, SOAP, and RMI calls. Theinstrumented entry points may include code that implements theprocessing of a received call or request, such as routines and methodsthat handle calls received by a virtual machine or application residingon an application server.

An application's object code, or bytecode, may be instrumented to insert“hooks”—portions of code that may retrieve information from anapplication, virtual machine, or other code. For example, applicationobject code or source code may also be modified or instrumented. Thehooks may be added via instrumentation to detect activity initiated byone more threads used by an application or virtual machine. The hooksmay retrieve information and send information without modifying thelogic of an application.

In some embodiments, byte instrumented code may detect a receivedrequest or call and identify the thread which is automaticallyassociated with the call. The thread identification is then provided toan agent, which may record the time of the received request. The agentmay also modify information associated with the thread as discussed inmore detail below. Instrumented byte code may also detect a call made byan application or virtual machine. When an outgoing call is received,the agent may record the time of the outgoing call as well as the threadthat initiated the call.

Agents may be installed on an application server at step 320. The agentsmay be installed on an application server and within a virtual machine,within an application, or outside a virtual machine. The agent may beadded by byte code instrumentation, by downloading code to be installedon to the application server, or by some other method. At some point,controller 190 may also be configured. Configuring controller 190 mayinclude loading software onto controller 190 for communicating with oneor more agents, processing runtime data, reporting performanceinformation, and performing other operations. Operation of controller190 is discussed in more detail with respect to FIG. 8.

The present technology may map and monitor a business transaction bycollecting data associated with calls received by and sent by anapplication or virtual machine. When a call is sent from one applicationto another, the present technology may modify the header of the callwith monitoring parameters, including an identifier of the source of thecall and the recipient of the call. Though calls may be received andsent in any order, steps 330-350 relate to processing a call received byan application and step 360 relates to processing a call sent by anapplication.

A request is received by an application server at step 330. The requestmay be received, such as for example, by application server 130 vianetwork server 125. The request may be received from an externalservice, such as from VM2 on application serer 140. A request may alsobe received by any of application servers 140-160 from anotherapplication server as part of a distributed business transaction. Next,the received request is associated with a thread by an agent at step340. The agent located on the application server which received therequest associates the request with a thread. Associating the requestwith a thread may include detecting the request at an instrumented entrypoint and identifying what business transaction is associated with therequest. Once the business transaction is identified, the businesstransaction is associated with the thread handling the request.Associating a request with a thread is discussed in more detail belowwith respect to the method of FIG. 4.

The received request may be processed at step 350. Processing therequest may include performing one or more operations or transactions byan application residing on the application server which received therequest.

When a request or call is received by an application or virtual machine,the present technology may insert an identifier for the recipient of thecall in the request, for example in the header of the received request.When receiving a request, monitoring parameters within the request mayindicate whether the call recipient was recognized by the callingentity. An agent in the recipient application server may determine thestatus of the recipient of the request (for example, whether theapplication receiving the call was known or unknown to the callingapplication) and proceed accordingly. For example, the agent on thereceiving application server may append or modify a portion ofmonitoring parameter, such as for example a call chain, and store theparameters locally. The agent may also verify its identity to thecontroller through one or more communications with controller 190.Processing a received request (or call) is discussed in more detailbelow with respect to FIG. 5.

A call to an external service may be detected at step 360. The call maybe required to complete processing of the request received at step 330.The call itself may be detected by instrumented exit points 260 or 270and may be made to an external service such as that provided by avirtual machine on an external application server.

When detected, the call may be modified with monitoring parameters. Anagent on the application server making the call may modify the call aspart of a business transaction. The agent may modify the call withmonitoring parameters, such as for example an application identifier,transaction identifier, request identifier, caller chain information,and diagnostic status. In some embodiments, the call is modified byadding thread information such as monitoring parameters from a “threadlocal” file to the outgoing thread. The monitoring parameter data may beadded to the “thread local” file by an agent. Generating a call inresponse to a received request is discussed in more detail below withrespect to FIG. 6.

An application server may respond to a received request at step 370. Ifa call is made by the application server while processing the request,the response to the call may be received and processed as part ofgenerating a response to the received request. Responding to a requestis discussed in more detail with respect to the method of FIG. 7A.

Runtime data may be reported to a controller at step 380. Each agent maycollect runtime data from instrumented entry points and exit pointsduring execution of applications within the virtual machine. As theagent receives the runtime data, the data may be aggregated and reportedto controller 190. Data, such as for example detailed data regarding aparticular request, may also be reported to controller 190 withoutaggregating the data. Reporting runtime data to a controller isdiscussed in more detail below with respect to FIG. 7B.

FIG. 4 is a flowchart of an exemplary method for associating a requestwith a thread. The method of FIG. 4 may provide more detail for step 340in the method of FIG. 3. A request may be associated with a businesstransaction at step 410. When a request is received by a virtualmachine, the instrumented entry point may detect the received call andreport the name of the call to an agent. The agent may determine whatbusiness transaction, if any, is associated with the received request.For example, the agent may compare the call name to a table of callnames and associated business transactions. The thread associated withthe request is identified at step 420. When a request is received, therequest is assigned to a thread. The identified thread will handle therequest until the request is completed.

The identified thread is then configured with monitoring parameterinformation at step 430. After determining that the request isassociated with a business transaction, and then identifying whichthread is associated with the request, the agent may configure thethread with monitor parameter information for the business transaction.The monitor parameter information may be added to a “thread local”memory for the thread handling the request. The monitoring parametersmay include an application identifier, transaction identifier, requestidentifier, call chain data, and diagnostics status.

The application identifier may be a global unique identifier (GUID) thatuniquely identifies the application handling the thread. The transactionidentifier may identify the business transaction associated with therequest. The business transaction may be identified at step 410. Arequest identifier may identifier the particular request received by theapplication or virtual machine. The call chain data may identify thechain of applications or virtual machines that have processed thecurrent business transaction thus far. For example, call chain data fora request received by VM4 from VM3 in the system of FIG. 1 may be“VM1-VM3-VM4.” The diagnostic status may indicate the level the currentbusiness transaction data should be collected and reported.

FIG. 5 is a flowchart of an exemplary method for processing a receivedrequest. The method of FIG. 5 may provide more detail for step 350 inthe method of FIG. 3. A determination is made as to the status of areceived at step 510. A request status may indicate the request isasynchronous, that the request is sent to a known external service, orthat the request is sent to an unknown external service.

The request may be asynchronous if a response is not expected by thedevice which made the call. For example, in system 100 of FIG. 1,application server 150 may send an asynchronous request to asynchronousnetwork machine 170. Rather than responding to application server 150,asynchronous network machine 170 may send a message to applicationserver 160.

If the received request is asynchronous at step 510, method of FIG. 5continues to step 525 where a service identifier is appended to the callchain. The service identifier may be added to the call chain by theagent associated with the virtual machine (or application) at theapplication server which receives the asynchronous message fromasynchronous network machine 170. The service identifier may be addedafter the previous device identifier in the call chain. Hence, a serviceidentifier for virtual machine 162 may be added after that forasynchronous network machine 170. Hence, if the call chain in system 100of FIG. 1 reads as “VM1-VM2”, the call chain may be appended with anservice identifier for a asynchronous network machine 170 such that thecall chain would read VM1-VM2-MQ when the message is received by aasynchronous network machine 170. This call chain would then be appendedto include VM4 when application server 160 received the asynchronouscommunication from asynchronous network machine 170. After appending theservice identifier to the previous device identifier in the call chain,the method of FIG. 5 continues to step 530.

If the request is made to a known service, the method continues to step530.

If the calling application or virtual machine did not recognize theexternal service to receive a request or call, the calling applicationmay place an unknown identifier at the end of the call chain in theheader of the request. Upon receiving the request and detecting theunknown service identifier in the call chain, the unknown recipient maytransmit a service identity verification message to controller 190. Theidentity verification message indicates to the controller that theservice received a request with a particular unknown service identifier.The controller may process the identity verification message asdiscussed in more detail with respect to the method of FIG. 8. Therecipient application may leave the unknown service identifier in thecall chain, and add to the call chain appropriately when a call isdetected that is related to the call chain (for example, by the samethread handling the received request). After transmitting the serviceidentity verification message to controller 190, the method of claim 5continues to step 530.

For example, virtual machine 132 may send a request to virtual machine152, but agent 134 executing in virtual machine 132 may not recognizevirtual machine 152. Agent 134 may place an “unknown” recipientidentifier in the call chain of the request to virtual machine 152, aswell as locally within the thread handling the call to virtual machine152, to indicate the call recipient is not known. When the call isreceived by the recipient application server, the agent on the recipientapplication server may send an identity verification message tocontroller 190 at step 550. The identity verification message informsthe controller of the actual identify for the “unknown” identifier, forexample that unknown identifier “U45” is associated with virtual machine152. The controller 190 may receive the request, store an indicationthat “U45” is associated with “VM4”, and transmit an update to theagents in the system of FIG. 1 that virtual machine 152 is associatedwith a particular identifier (for example “VM4”).

Returning to method 350, the received request is processed at step 530.Processing the request may include performing operations or executingmethods as called in the received request, as well as placing calls toother external services. When the request has completed, a response isgenerated and transmitted to the calling service. Responding to areceived request is discussed with respect to FIG. 3.

FIG. 6 is a flowchart of an exemplary method for generating a call. Themethod of FIG. 6 may provide more detail for step 360 in the method ofFIG. 3 and may be performed by an agent, such as agent 134 (though anyagent on an application or virtual machine may implement the method ofFIG. 6).

An outgoing call to an external service may be detected by aninstrumented exit point at step 610. The instrumented exit point codemay inform the agent of the type of call being made, such as for examplethe call protocol, by what application the call is being made, therecipient service of the call, and a time stamp associated with thecall.

Next, a determination is made as to the status of the called externalservice (i.e., virtual machine or application executing on a virtualmachine) at step 615. The external service status may be that theexternal service is known, the external service is unknown, or that theexternal service is called as part of an asynchronous transaction. Theagent may have access to a list of recognized external services,(virtual machines, applications) and may compare the intended externalservice with known external service identifiers. If the external serviceis not recognized by the agent, the agent may create an unknown serviceidentifier at step 625. The unknown service identifier may be sent tocontroller 190 at step 625. Identification information may also be sentto controller 190, such as an identification of the external service,the application server, a virtual machine, and other identificationinformation.

The unknown service identifier may be inserted at the end of a callchain to be included within the call being made at step 635. Forexample, the unknown service identifier may be inserted into a callchain within the thread handling the call, and the call chain within thethread may be placed in the header of the call. The method of FIG. 6then continues to step 650.

Returning to step 615, if the external service to receive the call isknown, the service identifier which will receive the application call isappended to the call chain at step 520. The service identifier may beinserted into a call chain within the thread handling the call, and thecall chain within the thread may eventually be placed in the header ofthe call. The method of FIG. 6 then continues to step 650.

If the call to the external service is part of an asynchronousapplication at step 615, an asynchronous service identifier is generatedat step 645. The asynchronous service identifier is appended to the callchain, similarly to a service identifier, at step 645. The identifiermay indicate that the portion of the transaction between the applicationmaking the application call and the recipient external service isasynchronous. After appending the asynchronous service identifier to thecall chain, the method of FIG. 6 then continues to step 650.

The agent may add monitoring parameters to the outgoing external servicecall to the recipient application at step 650. The monitoring parametersmay include an application identifier, a business transactionidentifier, a request identifier, and a call chain. The identifiers mayeach be implemented as a globally unique identifier (GUID). The callchain indicates the portion of a business transaction chain handledlocally by an application server. The call chain may identify nodeswhich receive and/or send calls or requests. For example, in the systemof FIG. 1, a call chain for a request received by application 130 willlist a node associated with the servers which implement applicationserver 130 or virtual machine 132, e.g. VM1. If virtual machine 132sends a request to virtual machine 142 on application server 140 as partof the business transaction, the call chain will comprise VM1-VM2. IfVM2 then calls VM4 on application server 160, which in turn calls datastore 180, the call chain may be extended to VM1-VM2-VM4 once virtualmachine 162 receives the call from virtual machine 142. The call chainmay be extended to VM1-VM2-VM4-DB1 when virtual machine 162 calls datastore 180.

The monitoring parameters may also indicate a diagnostics status. Thediagnostics status may be expressed as a boolean variable and indicatethat more detail of monitoring information should be collected for aparticular request. In some embodiments, if a particular businesstransaction, either in part or entirely, is determined to be operatingless than optimally or not as expected, controller 190 may automaticallyconfigure agents involved in monitoring that business transaction tocollect more detailed data associated a request associated with thatbusiness transaction. In collecting more detailed data, the diagnosticsstatus boolean valve may be set to collect more data. When thediagnostics status boolean is set to “on”, each agent involved inmonitoring the particular business transaction may collect informationassociated with the business transaction request, including each methodcalled as part of the business transaction, and not aggregate the dataassociated with the business transaction request. Rather, the runtimedata monitored for the business transaction request is returned tocontroller 190; the runtime data associated with a business transactionbeing monitored in a diagnostics status “on” may not be aggregated.

A call with monitoring parameters is made to an external service(virtual machine or application executing on a virtual machine) at step660. The call may be sent with the monitoring parameters included in thecall, for example in the call header or some other portion of the call.

FIG. 7A is a flowchart of an exemplary method for responding to areceived request. The method of FIG. 7A may provide more information forstep 370 in the method of FIG. 3. A response may be received from anexternal service at step 710 for a call made to the external service.The response may not be received for the call if the call is stalled. Inthis case, the agent at the current virtual machine or application whichsent the call may determine, such as for example after a specific periodof time, that the business transaction has stalled and indicate this inthe runtime data appropriately.

A response is generated and sent for a received request at step 720. Ifa call is made by a virtual machine while processing the request, theresponse to the call may be received and processed as part of generatinga response to the received request. After sending the response, thethread handling the call is closed at step 730.

FIG. 7B is a flowchart of an exemplary method for reporting runtime datato a controller. Runtime data may be aggregated at step 740. The runtimedata collected by an agent may be aggregated based on monitoringparameters and averaged over a period of time, for example one minute.

Runtime data associated with the call may be stored as it is received.In some embodiments, the runtime data may indicate the response time forthe call to complete. The runtime data may include timing informationassociated with a business transaction, call chain and other parameterinformation, and other data. An agent may receive or retrieve atimestamp corresponding to the beginning and the end of an applicationcall, method call, and other operations. The time stamps may be storedwith a business transaction identifier, application identifier, callingchain, and optionally other data for the request within a threadhandling the call. Information may be cleared from the thread handlingthe call once the application server has completed processing of arequest. Once the call is completed, a response time may be generatedfor the overall call as well as intervening calls to other applications.

A runtime data reporting event may be detected at step 750. The runtimereporting event may be any of several events, for example the expirationof a timer, a state of one or more resources of the application serverreporting the runtime data, or another event. For example, an agent maybe configured to report data periodically every minute, or some othertime period. The agent may also adjust the reporting based on the loadon the application server on which it resides, for example by waiting toreport runtime data if not many processor cycles are available orreporting the runtime data more often is a large number of processingcycles are available.

Runtime data may then be transmitted to a controller 190 by an agent atstep 760. The transmitted runtime data may include the aggregatedruntime data determined at step 750. Runtime data may also includenon-aggregated data, such as for example detailed request data collectedduring a diagnostics status “on” mode. Runtime data may be transmittedto a controller 190 periodically, for example every minute, based on anevent such as a request from controller 190 or the end of a businesstransaction being monitored in detail, or some other event.

Controller 190 may receive data from one or more agents, process thedata, and provide monitoring information regarding the system beingmonitored. When installed onto an application server, controller 190 maybe initialized. Controller initialization may include loadinginformation for application servers, such as identification information,loading transaction data, and other information. FIG. 8 is a flowchartof an exemplary method for controlling business transaction monitoring.The method of FIG. 8 may be performed by controller 190 in the system ofFIG. 1.

Controller 190 may receive an unknown service identifier message from anagent at step 805. For example, if virtual machine 152 is to make a callto virtual machine 162 (or application) and agent 154 on virtual machine152 does not recognize the recipient virtual machine (or application),agent 154 may generate an unknown service identifier and send theidentifier to controller 190. The controller 190 may store machineidentifiers, both known and unknown, and associated call names used byapplication methods.

A service identity verification message may be received by a controllerfrom an agent at step 810. The service identity verification message maybe generated by an agent and sent at step 520 in the method of FIG. 5.Upon receiving the service identity verification message, controller 190may update the unknown service identifier with the received serviceidentifier at step 830. Updating the unknown service identifier mayinclude associating the unknown service identifier with the applicationserver from which the application identity verification message wasreceived, and sending a message with a service identifier to use for theapplication server to each agent.

Aggregated runtime data may be received from one or more agents at step835. The aggregated runtime data may be received periodically, basedupon an event, based upon load size of the data, or based on some othercriteria. The aggregated runtime data may indicate a businesstransaction, call chain data, time stamp data, and other data. Thebusiness transaction may be associated with the request received at step330. The call chain data of the aggregated data may include the callchain data received in the header of a request, if any, along with anidentifier of the application or virtual machine processing the request.Aggregated data may be sent for each call chain combination. Forexample, for VM3 of FIG. 1, data may be aggregated for businesstransaction portions associated with call chain of VM1-VM3, VM1-VM3-VM4,VM3-VM4, or some other call chain portion.

A call chain for business transactions may be constructed from thereceived aggregated data at step 840. The call chain may be constructedby connecting data associated with sections of a business transactionbased on call chain data in the received aggregated data. For example, abusiness transaction “Check-out” may involve communications from VM1 toVM2 to VM4 in FIG. 1. Each of agents 134, 144, and 164 may reportaggregated data to controller 190. Agent 134 associated with VM1 mayreport data including the business transaction identifier, the timestamps associated with the start and end of the transaction (thetransaction that begins with VM1 receiving a request and ends with VM1sending a response), and an identification of entire call chain:VM1-VM2-VM4. Agent 144 associated with VM2 may report dada including thebusiness transaction identifier, call chain data associated of VM1-VM2,and time stamp data associated with receiving a request from VM1,sending a request to VM4, receiving a response from VM4, and sending aresponse to VM1. Agent 164 associated with VM4 may report dada includingthe business transaction identifier, call chain data associated ofVM3-VM4, and time stamp data associated with receiving a request fromVM3, and sending a response to VM3. The information received from eachagent for the identified business transaction may be used to generate amap of the transaction over different virtual machines. In this manner,the topology traversed by a business transaction can be determined by acontroller without any prior knowledge. An example of the mapping of abusiness transaction is illustrated in the interfaces of FIGS. 9 and 10.

Performance information may be determined for the business transactionat step 845. The performance information may indicate the total responsetime for the business transaction and local response times by each node(e.g., processing time by each application server or virtual machine inthe business transaction), as well as time periods between virtualmachines within the system, as well as whether the performance wasacceptable or unacceptable. For clusters representing a particularvirtual machine, the aggregated data may be averaged together by thecontroller 190.

Performance baselines and alerts may be determined for businesstransactions based on the determined performance at step 850. In someembodiments, an average or baseline performance may be determined for asection of a business transaction, for example by averaging performancedata for each section over a period of time. Once a baseline isdetermined, subsequent data can be compared to the baseline to determineif it is within a particular threshold based on the baseline. Thethreshold may be a predetermined percentage, such as 10%, the baselineitself, or some other value. Alternatively, a baseline and/orperformance threshold may be determined manually or in some othermanner. If performance data does not satisfy the threshold, an alert maybe generated and reported to an administrator.

The performance may be reported for a business transaction at step 855.For example, the performance may be reported through an interface suchas that shown in FIG. 9. After determining alerts or reporting theperformance for business transactions, controller 190 may automaticallymonitor individual requests based on the business transactionperformance at step 860. Automatically monitoring individual requestsmay include indicating to one or more agents that a particular requestshould be associated with a diagnostics status of “on.”

FIG. 9 is an exemplary interface for viewing monitoring data forbusiness transactions. In some embodiments, the interface of FIG. 9 canbe provided by controller 190 as part of a web service provided overnetwork server 125. The interface of FIG. 9 includes three monitoredvirtual machines 910, 920, and 940. The monitored system also includesmessage queue 930 and databases 950, 960 and 970. Agents located at themonitored virtual machines 910, 920 and 940 collect data and provide theaggregated runtime data to a controller such that the businesstransaction can be re-created as indicated in interface 900. Asindicated, between virtual machines 910 and 920, four calls per minutewere made from virtual machine 910 to virtual machine 920. Virtualmachine 920 made approximately four calls per minute to database 950 and26 calls per minute to database 960.

Interface 900 may be generated within a few minutes of initiating themonitoring of a particular system. By constructing the chain of abusiness transaction between monitored virtual machines, and associatingthe performance of each part of the chain, the application flow map suchas that shown in interface 900 may be generated easily and quicklycompared with other systems.

FIG. 10 is an exemplary interface for viewing monitoring data forbusiness transactions. FIG. 10 illustrates the same virtual machinearchitecture as illustrated in FIG. 9. The information displayed for thevirtual machines is associated with an application named “ACME OnlineBookstore” and a business transaction of “Checkout” as indicated justabove the interface. The interface of FIG. 9 includes three monitoredvirtual machines 910, 920, and 940, message queue 930 and databases 950,960 and 970. The calls sent as part of the business transaction arelabeled in representative communication lines between the machines aswell as the time to process each call. For example, the businesstransaction “Checkout” included a JMS call from virtual machine 910 tomessage queue 930. The call from machine 910 to machine 930 took anaverage of 6 milliseconds. Also included in interface 1000 is anindication of a load history, average response time history, requestsummary, and other data.

FIG. 11 illustrates an exemplary computing system 1100 that may be usedto implement an embodiment of the present invention. System 1100 of FIG.11 may be implemented in the contexts of the likes of data store 110,application server 120, network server 130, database 122, and clients150-160. The computing system 1100 of FIG. 11 includes one or moreprocessors 1110 and memory 1110. Main memory 1110 stores, in part,instructions and data for execution by processor 1110. Main memory 1110can store the executable code when in operation. The system 1100 of FIG.11 further includes a mass storage device 1130, portable storage mediumdrive(s) 1140, output devices 1150, user input devices 1160, a graphicsdisplay 1170, and peripheral devices 1180.

The components shown in FIG. 11 are depicted as being connected via asingle bus 1190. However, the components may be connected through one ormore data transport means. For example, processor unit 1110 and mainmemory 1110 may be connected via a local microprocessor bus, and themass storage device 1130, peripheral device(s) 1180, portable storagedevice 1140, and display system 1170 may be connected via one or moreinput/output (I/O) buses.

Mass storage device 1130, which may be implemented with a magnetic diskdrive or an optical disk drive, is a non-volatile storage device forstoring data and instructions for use by processor unit 1110. Massstorage device 1130 can store the system software for implementingembodiments of the present invention for purposes of loading thatsoftware into main memory 1110.

Portable storage device 1140 operates in conjunction with a portablenon-volatile storage medium, such as a floppy disk, compact disk orDigital video disc, to input and output data and code to and from thecomputer system 1100 of FIG. 11. The system software for implementingembodiments of the present invention may be stored on such a portablemedium and input to the computer system 1100 via the portable storagedevice 1140.

Input devices 1160 provide a portion of a user interface. Input devices1160 may include an alpha-numeric keypad, such as a keyboard, forinputting alpha-numeric and other information, or a pointing device,such as a mouse, a trackball, stylus, or cursor direction keys.Additionally, the system 1100 as shown in FIG. 11 includes outputdevices 1150. Examples of suitable output devices include speakers,printers, network interfaces, and monitors.

Display system 1170 may include a liquid crystal display (LCD) or othersuitable display device. Display system 1170 receives textual andgraphical information, and processes the information for output to thedisplay device.

Peripherals 1180 may include any type of computer support device to addadditional functionality to the computer system. For example, peripheraldevice(s) 1180 may include a modem or a router.

The components contained in the computer system 1100 of FIG. 11 arethose typically found in computer systems that may be suitable for usewith embodiments of the present invention and are intended to representa broad category of such computer components that are well known in theart. Thus, the computer system 1100 of FIG. 11 can be a personalcomputer, hand held computing device, telephone, mobile computingdevice, workstation, server, minicomputer, mainframe computer, or anyother computing device. The computer can also include different busconfigurations, networked platforms, multi-processor platforms, etc.Various operating systems can be used including Unix, Linux, Windows,Macintosh OS, Palm OS, and other suitable operating systems.

The foregoing detailed description of the technology herein has beenpresented for purposes of illustration and description. It is notintended to be exhaustive or to limit the technology to the precise formdisclosed. Many modifications and variations are possible in light ofthe above teaching. The described embodiments were chosen in order tobest explain the principles of the technology and its practicalapplication to thereby enable others skilled in the art to best utilizethe technology in various embodiments and with various modifications asare suited to the particular use contemplated. It is intended that thescope of the technology be defined by the claims appended hereto.

What is claimed is:
 1. A method for monitoring a business transaction,comprising: associating by an agent a received request with a threadthat handles the request; associating the thread with a monitoringparameter associated with the request, the monitoring parameterassociated with the business transaction; modifying thread data with asection of a call chain for a business transaction, the call chainidentifying two or more computers or services that have processed thebusiness transaction which includes the received request, wherein afirst agent on a first computer or service of the two or more computersor services adds an identifier to the call chain after the request isreceived by the first computer or service and a second agent on a secondcomputer or service of the two or more computers or services adds anidentifier to the call chain after the request is received by the secondcomputer or service; modifying an external request with the monitoringparameter from thread data by the agent, the call associated with thethread; and reporting runtime data associated with the received requestto a first server, the runtime data including the monitoring parameterand the call chain.
 2. The method of claim 1, further comprisinginstrumenting bytecode of the application to modify exit point or entrypoints of the application.
 3. The method of claim 1, wherein modifying arequest and reporting runtime data are performed by an agent.
 4. Themethod of claim 1, further comprising: detecting call chain data withinthe request; modifying the call chain data with an identifier associatedwith the application; and reporting the modified call chain data to thecontroller.
 5. The method of claim 1, further comprising: aggregatingdata for a business transaction; and reporting aggregated runtime dataassociated to the first server.
 6. The method of claim 1, furthercomprising: detecting an identifier for an unknown recipient in therequest; and modifying the unknown recipient identifier.
 7. The methodof claim 1, wherein the business transaction is distributed over one ormore servers.
 8. The method of claim 1, wherein the thread is associatedwith an application executing on a server.
 9. The method of claim 1,further comprising modifying an exit point or entry point of theapplication by embedding code, wherein the agent detects a receivedrequest through the entry point and detects a sent request through theexit point.
 10. The method of claim 1, further comprising: receiving afirst parameter from the first computer or service by a server;receiving a second parameter from the second computer or service by theserver; and correlating a distributed application processed on the firstcomputer and the second computer based on the first parameter and thesecond parameter.
 11. The method of claim 1, further comprising theexternal request includes placing a message in an asynchronous queue;associating the message with a business transaction; retrieving themessage from the asynchrounous queue by a third computer; andtransmitting business transaction information associated with themessage to a server by the third computer.
 12. The method of claim 1,further comprising: detecting by the first computer or service theexternal request to a second computer or service; sending the requestand a first name to the second computer or service by the first computeror service; transmitting the first name and request information to thefirst server by the first computer or service; and transmitting thefirst name and a second computer identifier to the first server by thesecond computer or service.
 13. The method of claim 12, furthercomprising: detecting by the second computer or service that the firstcomputer or service does not recognize the second computer or servicebased on the first name received by the second computer, wherein thefirst name and second computer or service identifier transmitted by thesecond computer or service is in response to the detection by the secondcomputer.
 14. A non-transitory computer readable storage medium havingembodied thereon a program, the program being executable by a processorto perform a method for monitoring a business transaction, the methodcomprising: associating by an agent a received request with a threadthat handles the request; associating the thread with a monitoringparameter associated with the request, the monitoring parameterassociated with the business transaction; modifying thread data with asection of a call chain for a business transaction, the call chainidentifying two or more computers or services that have processed thebusiness transaction which includes the received request, wherein afirst agent on a first computer or service of the two or more computersor services adds an identifier to the call chain after the request isreceived by the first computer or service and a second agent on a secondcomputer or service of the two or more computers or services adds anidentifier to the call chain after the request is received by the secondcomputer or service; modifying an external request with the monitoringparameter from thread data by the agent, the call associated with thethread; and reporting runtime data associated with the received requestto a first server, the runtime data including the monitoring parameterand the call chain.
 15. The non-transitory computer readable storagemedium of claim 14, the method further comprising instrumenting bytecodeof the application to modify exit point or entry points of theapplication.
 16. The non-transitory computer readable storage medium ofclaim 14, wherein modifying a request and reporting runtime data areperformed by an agent.
 17. The non-transitory computer readable storagemedium of claim 14, the method further comprising: detecting call chaindata within the request; modifying the call chain data with anidentifier associated with the application; and reporting the modifiedcall chain data to the controller.
 18. The non-transitory computerreadable storage medium of claim 14, the method further comprising:aggregating data for a business transaction; and reporting aggregatedruntime data associated to the controller.
 19. The non-transitorycomputer readable storage medium of claim 14, the method furthercomprising: detecting an identifier for an unknown recipient in therequest; and modifying the unknown recipient identifier.
 20. Thenon-transitory computer readable storage medium of claim 14, wherein thebusiness transaction is distributed over one or more servers.
 21. Thenon-transitory computer readable storage medium of claim 14, wherein thethread is associated with an application executing on a server.
 22. Thenon-transitory computer readable storage medium of claim 14, the methodfurther comprising modifying an exit point or entry point of theapplication by embedding code, wherein the agent detects a receivedrequest through the entry point and detects a sent request through theexit point.
 23. The non-transitory computer readable storage medium ofclaim 14, the method further comprising: receiving a first parameterfrom the first computer or service by a server; receiving a secondparameter from the second computer or service by the server; andcorrelating a distributed application processed on the first computer orservice and the second computer or service based on the first parameterand the second parameter.
 24. The non-transitory computer readablestorage medium of claim 14, the method further comprising the externalrequest includes placing a message in an asynchronous queue; associatingthe message with a business transaction; retrieving the message from theasynchrounous queue by a third computer; and transmitting businesstransaction information associated with the message to a server by thethird computer.
 25. The non-transitory computer readable storage mediumof claim 14, the method further comprising: detecting by a firstcomputer the external request to the second computer or service; sendingthe request and a first name to the second computer or service by thefirst computer; transmitting the first name and request information tothe first server by the first computer; and transmitting the first nameand a second computer identifier to the first server by the secondcomputer.
 26. The non-transitory computer readable storage of claim 25,the method further comprising: detecting by the second computer orservice that the first computer or service does not recognize the secondcomputer or service based on the first name received by the secondcomputer or service, wherein the first name and second computeridentifier transmitted by the second computer or service is in responseto the detection by the second computer.
 27. A system for monitoring abusiness transaction, comprising: a processor; a memory; one or moremodules stored in memory and executed by the processor, the one or moremodules executable to associate by an agent a received request with athread that handles the request, associate the thread with a monitoringparameter associated with the request, the monitoring parameterassociated with the business transaction, modify thread data with asection of a call chain for a business transaction, the call chainidentifying two or more computers or services that have processed thebusiness transaction which includes the received request, wherein afirst agent on a first computer or service of the two or more computersor services adds an identifier to the call chain after the request isreceived by the first computer or service and a second agent on a secondcomputer or service of the two or more computers or services adds anidentifier to the call chain after the request is received by the secondcomputer or service, modify an external request with the monitoringparameter from thread data by the agent, the call associated with thethread, and report runtime data associated with the received request toa first server, the runtime data including the monitoring parameter andthe call chain.
 28. The system of claim 27, wherein the businesstransaction is distributed over one or more servers.
 29. The system ofclaim 27, wherein the thread is associated with an application executingon a server.
 30. The system of claim 27, the one or more modulesexecutable by the processor to modify an exit point or entry point ofthe application by embedding code, wherein the agent detects a receivedrequest through the entry point and detects a sent request through theexit point.
 31. The system of claim 27, the one or more modulesexecutable to receive a first parameter from the first computer orservice by a server, receive a second parameter from the second computeror service by the server, and correlate a distributed applicationprocessed on the first computer or service and the second computer orservice based on the first parameter and the second parameter.
 32. Thesystem of claim 27, wherein the external request includes placing amessage in an asynchronous queue, associating the message with abusiness transaction, retrieving the message from the asynchrounousqueue by a third computer; and transmitting business transactioninformation associated with the message to a server by the thirdcomputer.
 33. The system of claim 27, the one or more modules executableto detect by the first computer or service the external request to thesecond computer or service, send the request and a first name to thesecond computer or service by the first computer or service, transmitthe first name and request information to the first server by the firstcomputer or service, and transmit the first name and a second computeridentifier to the first server by the second computer or service. 34.The system of claim 33, the one or more modules executable to detect bythe second computer or service that the first computer or service doesnot recognize the second computer or service based on the first namereceived by the second computer or service, wherein the first name andsecond computer identifier transmitted by the second computer or serviceis in response to the detection by the second computer or service.